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Measuring Organizational Conditions for Improving Quality 


Leaders, policymakers, and systems developers seek to improve early 
childhood programs through data driven decision-making. Data can 
be useful for informing continuous quality improvement efforts at the 
classroom and program level and for creating support for workforce 
development at the system level. Early childhood program leaders use 
assessments to help them understand their programs’ strengths and to 
draw attention to where supports are needed. 


Assessment data is particularly useful in understanding the complexity 
of organizational climate and the organizational conditions that lead to 
successful outcomes for children and families. Several tools are available 
for program leaders to assess organizational structures, processes, and 
workplace conditions, including: 


Preschool Program Quality Assessment (PQA)! 

Program Administration Scale (PAS)? 

Child Care Worker Job Stress Inventory (ECWJSI)$ 

Early Childhood Job Satisfaction Survey (ECJSS)* 

Early Childhood Work Environment Survey (ECWES)5 
Supportive Environmental Quality Underlying Adult Learning 
(SEQUAL)° 


The Early Education Essentials is a recently developed tool to examine 
program conditions that affect early childhood education instructional 
and emotional quality. It is patterned after the Five Essentials Framework,’ 
which is widely used to measure instructional supports in K-12 schools. 
The Early Education Essentials measures six dimensions of quality in 
early childhood programs: 


Effective instructional leaders 
Collaborative teachers 
Supportive environment 
Ambitious instruction 
Involved families 

Parent voice 
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A recently published validation study for the Farly Education Essentials 
demonstrates that it is a valid and reliable instrument that can be used 
to assess early childhood programs to improve teaching and learning 
outcomes. 


METHODOLOGY 

For this validation study, two sets of surveys were administered in 
one Midwestern city; one for teachers/staff in early childhood 
settings and one for parents/guardians of preschool-aged children. 
A stratified random sampling method was used to select sites with 
an oversampling for the percentage of children who spoke Spanish. 
The teacher surveys included 164 items within 26 scales and were 
made available online for a three-month period in the public schools. 
In community-based sites, data collectors administered the surveys to 
staff: Data collectors also administered the parent surveys in all sites. 


The parent survey was shorter, with 54 items within nine scales. Rasch 
analyses was used to combine items into scales. In addition to the 
surveys, administrative data were analyzed regarding school attendance. 
Classroom observational assessments were performed to measure 
teacher-child interactions. The Classroom Assessment Scoring System™ 
(CLASS)? was used to assess the interactions. 


Early Education Essentials surveys were analyzed from 8 1 early childhood 
program sites (41 school-based programs and 40 community-based 
programs), serving 3- and 4-year old children. Only publicly funded 
programs (e.g., state-funded preschool and/or Head Start) were included 
in the study. The average enrollment for the programs was 109 (sd = 64); 
91% of the children were from minority backgrounds; and 38% came 
from non-English speaking homes. Of the 746 teacher surveys collected, 
451 (61%) were from school-based sites and 294 (39%) were from 
community-based sites. There were 2,464 parent surveys collected (59% 
school; 41% community). About one-third of the parent surveys were 
conducted in Spanish. 


Data were analyzed to determine reliability, internal validity, group differences, 
and sensitivity across sites. Child outcome results were used to examine if 
positive scores on the surveys were related to desirable outcomes for children 
(attendance and teacher-child interactions). Hierarchical linear modeling 
(HLM) was used to compute average site-level CLASS scores to account for 
the shared variance among classrooms within the same school. Exploratory 
factor analysis was performed to group the scales. 


RESULTS 

The surveys performed well in the measurement characteristics 
of scale reliability, internal validity, differential item functioning, 
and sensitivity across sites. Reliability was measured for 25 scales 
with Rasch Person Reliability scores ranging from .73 to .92; with 
only two scales falling below the preferred .80 threshold. The Rasch 
analysis also provided assessment of internal validity showing that 
97% of the items fell in an acceptable range of >0.7 to <1.3 (infit 
mean squares). 


TheTeacher/ Staff survey could detect differences across sites, however 
the Parent Survey was less effective in detecting differences across sites. 
Differential item functioning (DIF) was used to compare if individual 
responses differed for school- versus community-based settings and 
primary language (English versus Spanish speakers). Results showed 
that 18 scales had no or only one large DIF on the Teacher/Staff Survey 
related to setting There were no large DIFs found related to setting on the 
Parent Survey and only one scale that had more than one large DIF related 
to primary language. The authors decided to leave the large DIF items in 
the scale because the number of large DIFs were minimal and they fit 
well with the various groups. 


The factor analysis aligned closely with the five essentials in the 


K-12 model. However researchers also identified a sixth factor—parent 
voice—which factored differently from involved families on the Parent 
Survey. Therefore, the Early Education Essentials have an additional 
dimension in contrast to the K-12 Five Essentials Framework. 


Outcomes related to CLASS scores were found for two of the six 
essential supports. Positive associations were found for Effective 
Instructional Leaders and Collaborative Teachers and all three of the 
CLASS domains (Emotional Support, Classroom Organization, and 
Instructional Support). Significant associations with CLASS scores 
were not found for the Supportive Environment, Involved Families, 
or Parent Voice essentials. Ambitious Instruction was not associated 
with any of the three domains of the CLASS scores. Table 1. HLM 
Coefficients Relating Essential Scores to CLASS Scores (Model 1) 
shows the results of the analysis showing these associations. 


TABLE 1. 
HLM Coefficients Relating Essential Scores to CLASS Scores (Model 1) 
Classroom Instructional 
Emotional Support Organization ‘Support 


B p- value B p- value B p- value 


Effective Instructional 


0.159 0.074 0.223 0.015 0190 0.025 
Leaders 


Collaborative Teachers 0.180 0.043 0.271 0.003 0.231 0.006 


Ambitious Instruction -0.149 0.086 = -0.099 0.279 = -0.130 ON7 


Supportive 


: 0.042 0.637 one 0.210 0.060 0.483 
Environment 


Involved Families 0.020 0.826 0.047 0.616 0.045 0.599 


Parent Voice -0.037 0.696 0.041 0.680 0.069 0.448 
NOTES: (1) Each essential coefficient comes from a separate model in which only that essential was included 


as a predictor. (2) Coefficients are presented in standardized terms. (3) The CLASS was standardized at the 
classroom level (within each domain) and then entered into the model. Model-estimated site-level SDs on 
the CLASS domains are (in standard terms): 0.70 (Emotional Support), 0.75 (Classroom Organization), and 
0.66 (Instructional Support). 

Outcomes related to student attendance were found for four of 
the six essential supports. Effective Instructional Leaders, Collaborative 
Teachers, Supportive Environment, and Involved Families were positively 
associated with student attendance. Ambitious Instruction and Parent 
Voice were not found to be associated with student attendance. The 
authors are continuing to examine and improve the tool to better measure 
developmentally appropriate instruction and to adapt the Parent Survey so 
that it will perform across sites. 


There are a few limitations to this study that should be considered. Since the 
research is based on correlations, the direction of the relationship between 
factors and organizational conditions is not evident. It is unknown whether 
the Early Education Essentials survey is detecting factors that affect outcomes 
(eg, engaged families or positive teacher-child interactions) or whether the 
organizational conditions predict these outcomes. This study was limited to 
one large city and a specific set of early childhood education settings. It has 
not been tested with early childhood centers that do not receive Head Start or 
state pre-K funding. 


DISCUSSION 

The Farly Education Essentials survey expands the capacity of early 
childhood program leaders, policymakers, systems developers, and 
researchers to assess organizational conditions that specifically affect 
instructional quality. Itis likely to be a useful tool for administrators seeking 
to evaluate the effects of their pedagogical leadership—one of the three 
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domains of whole leadership? When used with additional measures to 
assess whole leadership—administrative leadership, leadership essentials, as 
well as pedagogical leadership—stakeholders will be able to understand 
the organizational conditions and supports that positively impact child 
and family outcomes. Many quality initiatives focus on assessment at the 
classroom level, but examining quality with a wider lens at the site level 
expands the opportunity for sustainable change and improvement. The 
availability of valid and reliable instruments to assess the organizational 
structures, processes, and conditions within early childhood programs 
is necessary for data-driven improvement of programs as well as systems 
development and applied research. 


Findings from this validation study confirm that strong instructional 
leadership and teacher collaboration are good predictors of effective 
teaching and learning practices, evidenced in supportive teacher-child 
interactions and student attendance.'! This evidence is an important 
contribution to the growing body of knowledge to inform embedded 
continuous quality improvement efforts. It also suggests that leadership 
to support teacher collaboration like professional learning communities 
(PLCs) and communities of practice (CoPs) may have an effect on 
outcomes for children. 


This study raises questions for future research. The addition of the “parent 
voice” essential support should be further explored. If parent voice is an 
essential support why was it not related to CLASS scores or student attendance? 
With the introduction of the Farly Education Essentials survey to the existing 
battery of program assessment tools (PQA, PAS, ECWJSI, ECWES, ECJSS and 
SEQUAL), a concurrent validity study is needed to determine how these 
tools are related and how they can best be used to examine early childhood 
leadership from a whole leadership perspective. 
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